Image processing system and medical information processing system

ABSTRACT

In one embodiment, an image processing system includes a memory and processing circuitry. The memory is configured to store a predetermined program. The processing circuitry is configured, by executing the predetermined program, to perform processing on an input image by exploiting a neural network having an input layer, an output layer, and an intermediate layer provided between the input layer and the output layer, the input image being inputted to the input layer, and adjust an internal parameter based on data related to the input image, while performing the processing on the input image after training of the neural network, the internal parameter being at least one internal parameter of at least one node included in the intermediate layer, and the input parameter having been calculated by the training of the neural network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a division of and claims the benefit of priorityunder 35 U.S.C. § 120 from U.S. application Ser. No. 15/994,161 filedMay 31, 2018, and claims the benefit of priority under 35 U.S.C. § 119from Japanese Patent Application No. 2017-108934 filed Jun. 1, 2017, theentire contents of each of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processingsystem and a medical information processing system.

BACKGROUND

In recent years, a method of using a neural network has been used in thefield of various types of image processing and image recognition. Theneural network is a processing device in which one or more nodessimulating cells are connected by one or more edges that simulatenerves.

The neural network has biases and weighting coefficients, both of whichare associated with respective edges, as its internal parameters. Inaddition, each node has an activation function, and the parameters ofthe activation function are also included in the internal parameters ofthe neural network.

Machine learning is used for calculating the internal parameters of theneural network. For instance, in a neural network aimed at removingnoise of an image, the internal parameters are generally calculated inthe following manner.

First, initial setting is manually performed on various conditions suchas connection relationship between respective nodes, initial values ofthe biases and weighting coefficients associated with respective edges,and an initial value of the parameter of the activation function in eachnode.

Next, an image without noise (i.e., a “ground truth”) and a “trainingimage” in which noise is artificially superimposed on the ground truthare prepared. The initial parameters of the neural network aresequentially updated in such a manner that the difference between thetraining image processed in the neural network and the ground truthbecomes smaller. Such a series of processing flow is called “training”.

When difference, e.g., mean square error (MSE) of each pixel ofrespective images, between the ground truth and the image obtained byprocessing the training image in the neural network (i.e., the processedtraining image) becomes smaller than a predetermined value, the trainingwill end. The respective values of the internal parameters of the neuralnetwork at the end of training are held as internal parameterscalculated by training.

When using the neural network after the training, a noise-superimposedimage in which actual noise is superimposed is inputted to a neuralnetwork having internal parameters calculated by the training, and animage in which noise is removed or reduced is obtained at the output ofthe neural network.

In the neural network, high noise removal performance or noise reductionperformance is obtained, when the amount of superimposed noise in thenoise-superimposed image to be used is as large as the amount of noisein the training image, which has been used in the training process.However, in the neural network, the noise removal performance or noisereduction performance may deteriorate when the amount of noise in thenoise-superimposed image to be used is different from the amount ofnoise in the training image.

In order to cope with this problem, an approach may be considered inwhich plural training images, that have noise amounts different fromeach other, are used when training the internal parameters in the neuralnetwork, instead of using training images having a noise amount of asingle magnitude. However, in this approach, when assuming the noiseamount is same between the leaning process and the using process, thenoise removal or noise reduction performance may be reduced, comparedwith the noise removal or reduction performance obtained by using thetraining images that have a noise amount of a single magnitude.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a block diagram illustrating a configuration of an imageprocessing system of the first embodiment;

FIG. 2 is a schematic diagram illustrating an internal configuration ofa neural network in which a configuration capable of adjusting internalparameters is omitted;

FIG. 3 is a schematic diagram illustrating an internal configuration ofa node in which a configuration capable of adjusting internal parametersis omitted;

FIG. 4 is a schematic diagram illustrating connection relationshipbetween (a) the neural network of the present embodiment having aconfiguration capable of adjusting internal parameters and (b) anSNR-related data acquisition function and a parameter adjustmentfunction, both of which provides a control signal to the neural network;

FIG. 5 is a block diagram illustrating an internal configuration of onenode of the present embodiment;

FIG. 6A is a graph illustrating the Soft-Shrinkage function F_(SS)(x, T)as one case of an activation function used in the present embodiment;

FIG. 6B is a graph illustrating the Hard-Shrinkage function F_(HS)(x, T)as one case of an activation function used in the present embodiment;

FIG. 7 is a flowchart illustrating an operation of a learning mode;

FIG. 8 is a schematic diagram illustrating processing of a learning modein which a ground truth and a training image are used;

FIG. 9 is a flowchart illustrating processing of an image processingmode, after training performed by a neural network;

FIG. 10 is a schematic diagram illustrating a normalization processusing a maximum pixel value and a calculation method of a controlsignal;

FIG. 11A to FIG. 11C are schematic diagrams illustrating a case where athreshold value T of the Soft-Shrinkage function as an activationfunction is adjusted by the control signal G;

FIG. 12 is a schematic diagram illustrating an operation concept of aneural network according to the present embodiment in the imageprocessing mode;

FIG. 13 is a block diagram illustrating a configuration of the imageprocessing system according to the second embodiment;

FIG. 14 is a block diagram illustrating a configuration of the imageprocessing system according to the third embodiment; and

FIG. 15 is a block diagram illustrating a configuration of the imageprocessing system according to the fourth embodiment.

FIG. 16 is a block diagram illustrating a configuration of the imageprocessing system according to the fifth embodiment.

FIG. 17 is a block diagram illustrating a configuration of a medicalinformation processing system to which the image processing system isadapted.

DETAILED DESCRIPTION

Hereinafter, respective embodiments of image processing systems will bedescribed with reference to the accompanying drawings. In theembodiments described below, the same reference signs are given foridentical components in terms of configuration and function, andduplicate description is omitted.

In one embodiment, an image processing system includes a memory andprocessing circuitry. The memory is configured to store a predeterminedprogram. The processing circuitry is configured, by executing thepredetermined program, to perform processing on an input image byexploiting a neural network having an input layer, an output layer, andan intermediate layer provided between the input layer and the outputlayer, the input image being inputted to the input layer, and adjust aninternal parameter based on data related to the input image, whileperforming the processing on the input image after training of theneural network, the internal parameter being at least one internalparameter of at least one node included in the intermediate layer, andthe input parameter having been calculated by the training of the neuralnetwork.

First Embodiment

FIG. 1 is a block diagram illustrating an overall configuration of animage processing system 100 according to the first embodiment. The imageprocessing system 100 is constituted by, e.g., a computer system.

As shown in FIG. 1 , the image processing system 100 includes processingcircuitry 10, an input I/F (interface) 11, an output I/F (interface) 12,memory 21, a display 22, and an input device 23.

The processing circuitry 10 is, e.g., a circuit equipped with a centralprocessing unit (CPU) and/or a special-purpose or general-purposeprocessor. The processor implements various functions described below byexecuting programs stored in the memory 21. The processing circuitry 10may be configured as hardware such as a field programmable gate array(FPGA) or an application specific integrated circuit (ASIC). The variousfunctions described below can also be implemented by such hardware.Additionally, the processing circuitry 10 can implement the variousfunctions by combining hardware processing and software processing basedon its processor and programs.

The memory 21 is a recording medium including a read-only memory (ROM),a random access memory (RAM), and an external memory device such as ahard disk drive (HDD) or an optical disc device. The memory 21 storesvarious programs executed by a processor of the processing circuitry 10as well as various types of data and information.

The input device 23 includes various devices for an operator to inputvarious types of information and data, and is configured as, e.g., amouse, a keyboard, a trackball, and/or a touch panel.

The display 22 is a display device such as a liquid crystal displaypanel, a plasma display panel, and an organic EL panel.

The input I/F 11 and the output I/F 12 are interfaces for inputting andoutputting images, respectively. Each of the input I/F 11 and the outputI/F 12 may include various devices and circuits such as a wired orwireless LAN, various communication interfaces including a USB, anetwork interface including the Internet and a public telephone line,and a drive circuit of various storage media including an optical diskand a magnetic disk. By using these devices and circuits, the input I/F11 and the output I/F 12 can input and output image data, respectively.

As shown in FIG. 1 , the processing circuitry 10 implements a parameterlearning function 20, a neural-network processing function 30, anSNR-related data acquisition function 40, and a parameter adjustmentfunction 50 by, e.g., executing various programs stored in the memory21.

The parameter learning function 20 determines respective values ofinternal parameters of the neural network 60 by machine learning(hereinafter, simply referred to as learning) with the use of a trainingimage, as described below.

The neural-network processing function 30 performs a predeterminedprocessing on the input image inputted to the neural network 60 aftertraining, by using the neural network 60.

The SNR-related data acquisition function 40 acquires at least one ofsignal strength of the input image, magnitude of noise in the inputimage, signal-to-noise ratio (SNR) of the input image, gain used innormalization processing, data related to the signal-to-noise ratio,from the input image or supplementary information of the input image(hereinafter, the data acquired by the SNR-related data acquisitionfunction are referred to as “SNR-related data”). Note that theSNR-related data are data related to the input image.

The parameter adjustment function 50 adjusts the respective values ofthe internal parameters of the neural network 60 on the basis of theSNR-related data acquired by the SNR-related data acquisition function40, when the neural-network processing function 30 performs thepredetermined processing on the input image inputted to the neuralnetwork 60 after training.

As to the parameter learning function 20, the neural-network processingfunction 30, the SNR-related data acquisition function 40, and theparameter adjustment function 50, their details will be described below.

FIG. 2 is a schematic diagram illustrating an internal configuration ofthe neural network 60. Although the internal parameters are adjustablein the image processing system 100 according to the first embodiment,FIG. 2 illustrates only the basic configuration for avoidingcomplication and omits the configuration portion that can adjust theinternal parameters.

As shown in FIG. 2 , the neural network 60 has a multilayer structurethat includes an input layer 61, an output layer 63, and at least oneintermediate layer 62 between the input layer 61 and the output layer63.

The input layer 61 includes plural input terminals 70. The imageprocessing system 100 of the present embodiment performs imageprocessing such as noise reduction processing or noise removalprocessing by using the neural network 60. In this case, image data areinputted to the input layer 61. Further, for instance, pixel signalsconstituting image data are inputted to the respective input terminals70 of the input layer 61.

The output layer 63 also has plural output terminals 90. In the imageprocessing system 100 of the present embodiment, the image datasubjected to the image processing are outputted from the output layer63. For instance, pixel signals of the image data subjected to the imageprocessing are outputted to the respective output terminals 90 of theoutput layer 63.

Each of the intermediate layers 62 includes plural nodes 80. Focusing onone intermediate layer 62 between other intermediate layers 62, theoutput signals of the plural nodes 80 of the immediately precedingintermediate layer 62 are inputted to each node 80 of the focusedintermediate layer 62, and an output signal of each node 80 of thefocused intermediate layer 62 is distributed to plural nodes 80 of theimmediately subsequent intermediate layer 62.

When the intermediate layer 62 is immediately subsequent to the inputlayer 61, plural signals of the respective plural input terminals 70 areinputted to each node 80 of this intermediate layer 62, and each of theplural signals are outputted to the nodes 80 of the intermediate layer62 of the subsequent stage.

When the intermediate layer 62 is immediately prior to the output layer63, output signals of the respective nodes 80 of this intermediate layer62 are outputted to the output terminals 90 of the output layer 63

FIG. 3 is a schematic diagram illustrating an internal configuration ofthe node 80. Although FIG. 3 illustrates a node 80 in the intermediatelayer 62 that is immediately subsequent to the input layer 61, othernodes in the intermediate layer 62 may have the same configuration.Plural signals are inputted to the node 80 shown in FIG. 3 . Respectiveinput signals are subjected to weighting with weights 800 through themultipliers 801, and then summed up by an adder 802. Further, an adder804 adds a bias 803 to the output of the adder, and then, the output ofthe adder 804 is sent to the activation function 805. When the output ofadder 804 is defined as “x”, “x” is expressed by the following equation(1).

$\begin{matrix}{x = {\left( {\sum\limits_{i}{w_{i}s_{i}}} \right) + b}} & {{Equation}(1)}\end{matrix}$

In the equation (1), “w_(i)” is the weight 800, “s_(i)” is the inputsignal, “i” is the number of the input signal, and “b” is the bias 803.

By applying the activation function 805 to “x” in the equation (1), theoutput of the node 80 is determined. Various subdifferentiablefunctions, such as a clipping function expressed by the equation (2), aReLU function expressed by the equation (3), or a tanh functionexpressed by the equation (4), have been conventionally used.

$\begin{matrix}{{\text{?}\left( {x,U,L} \right)} = \left\{ {\begin{matrix}U & \left( {x > U} \right) \\L & \left( {x < L} \right) \\x & {otherwise}\end{matrix},{x \in R},{U \in R},{L \in R}} \right.} & {{Equation}(2)}\end{matrix}$ $\begin{matrix}{{f_{ReLU}(x)} = \left\{ \begin{matrix}0 & \left( {x < 0} \right) \\x & {otherwise}\end{matrix} \right.} & {{Equation}(3)}\end{matrix}$ $\begin{matrix}{{\tanh(x)} = \frac{e^{x} - e^{- x}}{e^{x} + e^{- x}}} & {{Equation}(4)}\end{matrix}$ ?indicates text missing or illegible when filed

In the present embodiment, for instance, the Soft-Shrinkage functionexpressed by the following equation (5) may be used as the activationfunction 805. The Soft-Shrinkage function will be described below.

$\begin{matrix}{{f_{ss}\left( {x,T} \right)} = \left\{ \begin{matrix}{x - T} & \left( {x > T} \right) \\{x + T} & {\left( {x < {- T}} \right),{T \geq 0}} \\x & {otherwise}\end{matrix} \right.} & {{Equation}(5)}\end{matrix}$

In addition, the internal parameters of the conventional neural network60 are determined by learning performed in advance. Then, whenexploiting the neural network 60, i.e., when using the neural network60, the respective determined values of the internal parameters are usedin a fixed state.

By contrast, in the present embodiment, the respective values of theinternal parameters of the neural network 60 can be adjusted on thebasis of data related to the input image, when predetermined processingis performed after training, i.e., when the neural network 60 isexploited after training.

FIG. 4 is a schematic diagram illustrating connection relationshipbetween (a) the neural network 60 of the present embodiment configuredto be able to adjust the internal parameters and (b) the parameteradjustment function 50 and the SNR-related data acquisition function 40providing a control signal to the neural network 60.

The parameter adjustment function 50 supplies the control signal to atleast some nodes 80 of the intermediate layers 62. For instance, theparameter adjustment function 50 supplies a control signal to the nodes80 of at least one intermediate layer 62. As shown in FIG. 4 , thecontrol signal may be supplied from the parameter adjustment function 50to each node 80 of the intermediate layers 62.

FIG. 5 is a block diagram illustrating an internal configuration of onenode 80 of the present embodiment. The node 80 shown in FIG. 5 isconfigured such that threshold T, which is the parameter of theactivation function 805 (i.e., one of the internal parameters of thenode 80), is adjusted with respect to the activation function 805.

FIG. 6A is a graph illustrating a Soft-shrinkage function F_(SS)(x, T)as an example of the activation function 805 in which the threshold T isused as a parameter. Similarly, FIG. 6B is a graph illustrating aHard-Shrinkage function F_(HS)(x, T) as an example of the activationfunction 805 in which the threshold T is used as a parameter. Each ofthe Soft-Shrinkage function F_(SS)(x, T) and the Hard-Shrinkage functionF_(HS)(x, T) is a function that outputs zero when an input value lieswithin a predetermined positive/negative threshold ±T around zero, andoutputs a value proportional to an input value when an input value liesoutside the range of the threshold ±T.

Note that, when the input is out of the range of the threshold ±T, theamplitude of the output of the Soft-Shrinkage function F_(SS)(x, T)becomes smaller than the amplitude “x” of the input by the threshold T.Meanwhile, the amplitude of the output of the Hard-Shrinkage functionF_(HS)(x, T) has the same value as the amplitude “x” of the input, whenthe input is out of the range of the threshold ±T,

By using the Soft-Shrinkage function F_(SS)(x, T) or the Hard-Shrinkagefunction F_(HS)(x, T) as the activation function 805, a signal withamplitude smaller than the threshold value T, i.e., a weak signal, whichmost likely be a noise, can be made zero at the output of the activationfunction.

As described above, the neural network 60, which uses the Soft-Shrinkagefunction F_(SS)(x, T) or the Hard-Shrinkage function F_(HS) (x, T) asthe activation function 805, is a technology for reducing noise in animage and exerts a noise reduction effect when the threshold value T isset to an appropriate magnitude with respect to the noise level.

As shown in FIG. 5 , the threshold value T can be determined by, e.g.,T=αG as the product of the coefficient α, which is the internalparameter, and the control signal G, which comes from the outside. Forinstance, the SNR-related data acquisition function 40 acquires datarelated to SNR or magnitude of noise of the input image (i.e.,SNR-related data) from the input image (or from its supplementaryinformation) inputted via the input I/F 11, and outputs the acquiredSNR-related data to the parameter adjustment function 50. Further, theparameter adjustment function 50 determines magnitude of the controlsignal G on the basis of the SNR-related data and supplies the controlsignal G to each node 80. For instance, when the SNR-related dataindicates that SNR of the input image is low or noise included in theinput image is large, the parameter adjustment function 50 sets thecontrol signal G to be a large value and supplies it to each of thenodes 80.

The image processing system 100 of the present embodiment is a systemthat exploits the neural network 60. Thus, the image processing system100 has “a learning mode” for causing the neural network 60 to learn and“an image processing mode” for performing image processing by exploitingthe neural network 60 after the training performed by the neural network60.

FIG. 7 is a flowchart illustrating an operation of the learning mode ofthe image processing system 100, out of the above-described twooperation modes.

First, in the step ST10, the control signal G from the outside to theneural network 60 is set to a fixed value. The control signal G shown inFIG. 5 is fixed to, e.g., G=1.

In the next step ST11, the respective values of the internal parametersof the neural network 60 are set to initial values (i.e., defaultvalues). For instance, the respective values of the internal parameterssuch as the value of the weight 800 in each node 80, the value of thebias 803, and the value of the coefficient α shown in FIG. 5 are set tothe initial values.

In the next step ST12, a ground truth IM is set. The image processingsystem 100 according to the present embodiment performs processing forremoving or reducing noise included in an input image. Thus, an imagewith no noise or an image with a very high SNR is selected as the groundtruth IM, and this ground truth IM is set as preparation for training inthe neural network 60.

In the next step ST13, a training image obtained by adding artificialnoise to the ground truth is generated, and similarly, this trainingimage is set as preparations for training in the neural network 60. Theartificial noise is, e.g., Gaussian noise that is generated bycalculation so as to have a standard deviation σ of a predeterminedmagnitude, and the training image is generated by adding this artificialnoise to each pixel value of the ground truth.

In the next step ST14, the generated training image is inputted to theneural network 60, and its processing result is obtained.

For instance, the pixel values of the respective pixels of the trainingimage, to which noise is added, are inputted to the respective inputterminals 70 of the input layer 61 of the neural network 60. These pixelvalues propagate each node 80 of the intermediate layers 62 of theneural network 60 from the input layer 61 to the output layer 63 whilechanging their values by being subjected to weighted addition, biasaddition, and activation function processing shown in FIG. 5 , and thenoutputted to the output terminals 90 of the output layer 63 as the pixelvalues of the image processed by the neural network 60.

In next step ST15, an error (i.e., training error) between the processedimage and the ground truth is calculated. Specifically, as the trainingerror, it is possible to use the mean square error (MSE) and the sum ofsquare errors, both of which can be obtained from each pixel of theprocessed image and each pixel of the ground truth.

In the next step ST16, the parameters of the neural network are updatedsuch that the training error becomes smaller by using the error backpropagation algorithm. In particular, the algorithms stated in thefollowing Document 1 and Document 2 can be used for updating theparameters of the Shrinkage function.

-   [Document 1] Z. Wang et.al, “Deep Networks for Image Super    Resolution with Sparse Prior,” Proc. ICCV2015.-   [Document 2] X. Zhang, “Thresholding Neural Network for Adaptive    Noise Reduction,” IEEE Transactions on Neural Networks, vol. 12, no.    3, 2001.

In the next step ST17, by using the correct image that is an imagehaving a high SNR prepared separately from the ground truth and averification image obtained by adding noise to the correct image, theparameter learning function 20 calculates an error (generalizationerror) between the correct image and an image obtained by processing theverification image, using the neural network 60.

In the next step ST18, it is determined whether the generalization errorhas reached the local minimum value or not, and the parameter updatingis completed when the generalization error reaches the local minimumvalue. Conversely, when the generalization error does not reach thelocal minimum value, the processing returns to the step ST12 and theprocessing is repeated from the step ST12.

FIG. 8 is a schematic diagram illustrating the above-describedprocessing. As shown in the upper part of FIG. 8 , the ground truth IMahaving no noise or a very high SNR is set on the output side of theneural network 60. In addition, a training image obtained by addingartificial noise Na to this ground truth IMa is inputted to the neuralnetwork 60. Then, the internal parameters of the neural network 60 aredetermined in such a manner that the training image processed by theneural network 60 approaches the ground truth. The process ofdetermining these internal parameters is a process called machinelearning or simply learning.

Although the number of the ground truth(s) used for training may be one,the training effect can be improved by causing the neural network 60 tolearn by using plural similar different ground truth. For instance, asshown in the middle part and the bottom part of FIG. 8 , the trainingeffect can be enhanced by training with plural images in which thecombination of the ground truth IMb and the training image IMb+Nb,and/or the combination of the ground truth IMc and the training imageIMc+Nc are used in addition to the combination of the ground truth IMaand the training image IMa+Na.

In the case of training with the use of plural ground truth, first, thesteps ST12 to ST18 in the flowchart of FIG. 7 are executed for selectedone of the plural ground truth. In this case, when it is determined thatthe generalization error has not reached the local minimum value forthis selected ground truth in the step ST18, the processing returns tothe step ST12, in which the ground truth is replaced by a new groundtruth (i.e., another of the plural ground truth) so that the processingfrom the steps ST13 to ST18 is executed for the replaced new groundtruth.

When training of all the ground truth has been completed, the internalparameters of the neural network 60 after the training are stored in thememory 21 in the step ST19. Alternatively or additionally, informationregarding the structure of the neural network 60 may be stored in thememory 21. That is, assuming that the structure of the neural network 60and the internal parameters are referred to as a machine learning model,in the step ST19, the entire machine learning model after training maybe stored.

Each process of the above-described learning mode is performed by theparameter learning function 20 in FIG. 1 .

FIG. 9 is a flowchart illustrating processing of the image processingmode of the image processing system 100, i.e., an operation mode at thetime of exploiting the neural network 60 after completion of training bythe neural network 60.

In the first step ST20, an image is inputted to the processing circuitry10 via the input I/F 11. Specifically, an image is inputted to theSNR-related data acquisition function 40. The image to be inputted is animage to be subjected to noise removal processing or noise reductionprocessing with the use of the neural network 60. For instance, theinput image is an image imaged by a medical image diagnostic apparatussuch as an MRI (magnetic resonance imaging) apparatus, a CT(computerized tomography) apparatus, and an ultrasonic diagnosticapparatus. The SNR-related data are attached to this image.

In the next step ST21, the SNR-related data acquisition function 40acquires the SNR-related data from the input image.

In the next step ST22, the parameter adjustment function 50 generatesthe control signal for each of the nodes 80 of the neural network 60from the SNR-related data.

There are various specific methods to acquire the SNR-related data froman input image to generate the control signal.

FIG. 10 is a schematic diagram illustrating a method of calculating thecontrol signal G, when normalization of each pixel value by using themaximum pixel value Lv in the image is performed before the noiseremoval processing or noise reduction processing by the neural network60.

When such normalization processing is performed, the SNR-related dataacquired by the SNR-related data acquisition function 40 correspond tothe maximum pixel value Lv in the image. For instance, as shown in theleft column of FIG. 10 , it is assumed that processing targets are threetypes of images and the respective maximum pixel values Lv of thesethree types of images are Lv1, Lv2, and Lv3, which are different fromeach other (Lv2>Lv1>Lv3). In this case, in order to make the maximumpixel values after normalization to become Smax, each pixel ofrespective three images is needed to be multiplied by the respectivegains of G1=Smax/Lv1, G2=Smax/Lv2, and G3=Smax/Lv3.

Usually, the magnitude (e.g., rms value of noise) of noise of an imageat the time of imaging (i.e., noise of the image before normalization)is common between images. Thus, when normalization is performed withrespect to the maximum pixel value Lv in the image, the magnitude N ofthe normalized noise differs for each image, as shown in the rightcolumn of FIG. 10 (N2<N1<N3). By providing the gain G used for thenormalization to the neural network 60 as the control signal, it ispossible to provide the neural network 60 with data that havecorrelation with the magnitude of the noise of the processing targetimage.

Returning to FIG. 9 , in the next step ST23, the internal parameters ofthe respective nodes are adjusted in accordance with the control signalprovided from the parameter adjustment function 50.

FIG. 11A to FIG. 11C are schematic diagrams illustrating a case wherethe internal parameter adjusted in the step ST23 is the threshold valueT of the Soft-Shrinkage function. Specifically, FIG. 11A illustrates acase where the activation function 805 is the Soft-Shrinkage functionexpressed by the equation (5) and the threshold T of the Soft-Shrinkagefunction is adjusted by the control signal G so as to become T1=αG1.Similarly, FIG. 11B and FIG. 11C illustrate cases where the activationfunction 805 is the same Soft-Shrinkage function and the threshold T ofthe Soft-Shrinkage function is adjusted by the control signal G so as tobecome T2=AG2 and T3=αG3, respectively.

As the gain G (i.e., magnitude of noise) increases, the width of theinput at which the output becomes zero also increases, and the effect ofremoving or reducing noise is maintained, independent of the magnitudeof the noise.

In the next step ST24, the image inputted to the input I/F 11 isinputted to the neural network 60.

In the next step ST25, the image inputted to the neural network 60 isprocessed in the neural network 60.

In the next step ST26, the image processed in the neural network 60 isoutputted from the output I/F 12.

FIG. 12 is a schematic diagram illustrating the operation concept of theneural network 60 of the present embodiment in the image processingmode. The image processing system 100 of the present embodiment has afunction of acquiring the SNR-related data from the input image IMin ofthe neural network 60 and adjusting the parameters of the neural network60 on the basis of the acquired SNR-related data.

Conventional neural networks have the problem that the effect ofremoving or reducing noise decreases when magnitude of noise included inthe image used in the learning mode is different in magnitude of noiseincluded in the image in the image processing mode after the training.

However, even after training of the neural network 60, the imageprocessing system 100 according to the present embodiment has a functionof adjusting the parameters of the neural network 60 depending on themagnitude of noise included in the image that is inputted in the imageprocessing mode. As a result, even when magnitude of noise included inthe image used in the learning mode is different in magnitude of noiseincluded in the image inputted in the image processing mode aftertraining, it is possible to maintain the effect of removing or reducingnoise.

The SNR-related data are not limited to the gain G used for normalizingan image with the maximum pixel value, as mentioned above. Hereinafter,some other examples of the SNR-related data will be described.

For instance, when the image to be processed is an image imaged by acamera, the SNR-related data acquisition function 40 may acquire anF-value and/or an exposure time of the camera at the time of imaging asthe SNR-related data from, e.g., supplementary information of the cameraimage. This is because the F value and the exposure time influence thenumber of photons made incident on the pixels of the image sensor andthe shot noise decreases as the number of photons increases.

For instance, when the image to be processed is a medical imagegenerated by an MRI apparatus, the SNR-related data acquisition function40 may acquire one or more combinations of parameters related to imagingconditions from the MRI apparatus as the SNR-related data. Theparameters related to imaging conditions are, e.g., number of phaseencoding steps Ny, number of frequency encoding steps Nx, voxel volume(ΔxΔyΔz), bandwidth Bw, and number of times of integration Nex. This isbecause SNR of a medical image generated by an MRI apparatus isexpressed by the following expression.

${SNR} \propto {\Delta x\Delta y\Delta z\sqrt{\frac{\text{?}}{B_{w}}}}$?indicates text missing or illegible when filed

Further, for instance, when the image to be processed is a medical imagegenerated by a CT apparatus, the SNR-related data acquisition function40 may acquire at least one of tube voltage, tube current, irradiationtime, object size, exposure dose, and/or the number of beams, from theCT apparatus as the SNR-related data. This is because these parameterscorrelate with SNR and noise amount of a medical image generated by a CTapparatus.

In addition, when the image to be processed is a radar image, theSNR-related data acquisition function 40 may acquire the magnitude ofthe received signal and/or the gain of the signal receiver as theSNR-related data.

Further, in the case of using a signal value such as Lv for adjustingthe threshold value T of the Soft-Shrinkage function, there is apossibility that the processing performance may deteriorate when dynamicrange of the signal changes. For instance, consider a case wheretraining is completed by using data of an input image in which dynamicrange of each pixel value corresponds to 8 bit, while the processingcircuitry 10 is connected to an image sensor (e.g., a camera) configuredto generate image data in which dynamic range of each pixel correspondsto 10 bit. In this case, since magnitude of noise becomes relativelysmaller as dynamic range of each pixel becomes larger, the value of thethreshold value T becomes smaller and the noise removal performanceconsequently deteriorates as shown by the comparison between FIG. 11Band FIG. 11C. In such a case, the SNR-related data acquisition function40 may acquire information on the dynamic range of the pixel value andmultiply the control signal G by the ratio between the dynamic range atthe time of imaging and the dynamic range at the time of training.

Moreover, the image processing system 100 may be configured such thatthe noise removal or noise reduction performance can be changed byuser's adjustment. For instance, a user sets the denoising strength(i.e., parameter value that changes degree of noise removal or noisereduction) via the input device 23 of the image processing system 100.The SNR-related data acquisition function 40 or the parameter adjustmentfunction 50 can change the noise removal or noise reduction performanceof the neural network 60 by acquiring the denoising strength having beenset by the user and multiplying the control signal G by the denoisingstrength.

Although FIG. 4 illustrates the case where the control signal isinputted from the parameter adjustment function 50 to every node 80 ineach of the intermediate layers 62, the image processing system 100 isnot necessarily limited to this configuration. The image processingsystem 100 may be configured such that the control signal is inputtedonly to some of the nodes 80. Alternatively, the image processing system100 may be configured such that only the parameters of some of the nodes80 are adjusted on the basis of the SNR-related data, while theparameters of the other nodes 80 are fixed to parameter valuesdetermined in the learning mode.

Although the Soft-shrinkage function F_(SS)(x, T) shown in FIG. 6A andexpressed by the equation (5), or the Hard-Shrinkage function F_(HS)(x,T) shown in FIG. 6B are exemplified as the activation functions used inthe present embodiment, various activation functions other than thesetwo functions can also be used for the neural network 60 of the presentembodiment.

For instance, a function using the tanh function expressed by thefollowing equation (6) may be used as the activation function.

$\begin{matrix}{{f\left( {x,T} \right)} = {x - {T{\tanh\left( \frac{x}{T} \right)}}}} & {{Equation}(6)}\end{matrix}$

Alternatively, a function using the clipping function f_(clip) expressedby the following equation (7) may be used as the activation function.

f(x,T)=x−f _(clip)(x,T,−T)  Equation (7)

Further alternatively, a function using the ReLU function f_(ReLU)expressed by the following equation (8) may be used as the activationfunction.

f(x,T)=f _(ReLu)(f _(ReLU)(x)−T)−f _(ReLU)(f _(ReLU)(−x)−T)  Equation(8)

Other Embodiments

Hereinafter, a description will be given of embodiments of the neuralnetwork 60 that has a configuration different from the configuration ofthe above-described first embodiment.

FIG. 13 is a block diagram illustrating a configuration of the imageprocessing system 100 according to the second embodiment. The blockdiagram of FIG. 13 shows the configuration focusing on the operation inthe image processing mode and in particular, the internal configurationof the neural network 60 related to the image processing mode. Althoughthe input device 23, the memory 21, the display 22, and the parameterlearning function 20 among all the components of the image processingsystem 100 shown in FIG. 1 are omitted in FIG. 13 , these components maybe included in the image processing system 100 according to the secondembodiment.

The neural network 60 of the second embodiment includes a neural network60 a for residual estimation, a path 807 bypassing the neural network 60a, and a subtractor 806 for subtracting the output (residual) of theneural network 60 a from the output of the path 807.

The neural network 60 a is a neural network generated by residuallearning. In this case, the neural network 60 a estimates only theresidual obtained by removing the signal value from each pixel value ofthe input image. That is to say, the neural network 60 a estimates onlythe noise component included in the input image. Then, the neuralnetwork 60 a outputs only the estimated noise component to one end ofthe subtractor 806.

Meanwhile, the input image including noise is inputted to the other endof the subtractor 806 through the path 807. In the subtractor 806, thenoise estimated by the neural network 60 a is subtracted from the inputimage. As a result, the subtractor 806, i.e., the neural network 60outputs such an image that noise is removed from the input image ornoise is reduced.

The neural network 60 of the second embodiment is also configured suchthat the internal parameters of the neural network 60 a can be adjustedfor residual estimation on the basis of the input image or theSNR-related data acquired from the supplementary information of theinput image. Thus, even when magnitude of noise in the image used in thelearning mode is different from that used in the image processing mode,deterioration of noise removal or noise reduction performance can beprevented.

FIG. 14 is a block diagram illustrating a configuration of the imageprocessing system 100 according to the third embodiment. The blockdiagram of FIG. 14 also illustrates the configuration focusing on theoperation in the image processing mode and in particular, the internalconfiguration of the neural network 60 related to the image processingmode, similarly to FIG. 13 .

The neural network 60 of the third embodiment is configured such thatthe input to the neural network 60 is divided into two paths to beprocessed. One path is provided with at least one adaptive-type neuralnetwork 60 b, and another path is provided with at least one fixed-typeneural network 60 c. The neural network 60 is further provided with anadder 808 for adding the respective outputs of the two neural networks60 b and 60 c.

In the image processing system 100 according to the third embodiment,only the adaptive-type neural network 60 b is configured to be able toadjust its internal parameters in the image processing mode on the basisof the SNR-related data acquired from the input image or thesupplementary information of the input image. In the meantime, theinternal parameters of the fixed-type neural network 60 c are fixed tothe values determined in the learning mode, even in the image processingmode.

Since noise is removed or reduced in the respective images outputtedfrom the neural network 60 b and the neural network 60 c, the output I/F12 outputs such an image that noise is reduced or removed from theinputted image, by adding the image outputted from the neural network 60b to the image outputted from the neural network 60 c.

Note that even in the third embodiment, the internal parameters of theneural network 60 b are adjusted on the basis of the SNR-related data.Thus, even when noise of the image used in the learning mode isdifferent from that used in the image processing mode, deterioration ofnoise removal or noise reduction performance can be prevented.

FIG. 15 is a block diagram illustrating a configuration of the imageprocessing system 100 according to the fourth embodiment. The blockdiagram of FIG. 15 also illustrates the configuration focusing on theoperation in the image processing mode, and in particular, the internalconfiguration of the neural network 60 related to the image processingmode, similarly to FIG. 13 . Although the input device 23, the memory21, the display 22, and the parameter learning function 20 shown in FIG.1 are omitted in FIG. 15 , these components may be included in the imageprocessing system 100 according to the fourth embodiment.

The neural network 60 of the fourth embodiment is configured to divide afrequency region of an input image into a high frequency region and alow frequency region to perform processing on signals of two (or plural)frequency bands. For instance, the neural network 60 of the fourthembodiment 60 includes a divider 809, at least one neural network 60 dfor high frequency, at least one neural network 60 e for low frequency,and an image composer 810.

The divider 809 converts the input image expressed in the real spaceregion into data in the spatial frequency region by performing, e.g.,two-dimensional Fourier transform on the input image. Further, thedivider 809 divides the transformed data in the spatial frequency regioninto, e.g., high frequency band data and low frequency band data byapplying, e.g., a high pass filter and/or a low pass filter.

Thereafter, the high frequency band data are inputted to the neuralnetwork 60 d for high frequency and subjected to image processing. Theneural network 60 d for high frequency may be configured as aninternal-parameter adaptive type. That is, the image processing system100 is configured such that the internal parameters can be adjusted onthe basis of the SNR-related data acquired from the input image or thesupplementary information of the input image even after training.

On the other hand, the low frequency band data are inputted to theneural network 60 e for low frequency and subjected to image processing.The neural network 60 e for low frequency may be configured as aninternal-parameter fixed type. That is, the internal parametersdetermined in the learning mode are fixed in the image processing mode,and the input data are processed in this state.

Since noise is removed or reduced in the respective images outputtedfrom the neural networks 60 d and 60 e, the image composer 810 composesthe respective images outputted from the neural networks 60 d and 60 eso as to cause the output I/F 12 to output such an image that noise isreduced or removed from the inputted image.

Also in the fourth embodiment, the internal parameters of the neuralnetwork 60 d are adjusted on the basis of the SNR-related data. Thus,even when magnitude of noise used in the learning mode is different fromthat used in the image processing mode, deterioration of the noiseremoval or noise reduction performance can be prevented.

FIG. 16 is a block diagram showing a configuration example of the imageprocessing system 100 according to the fifth embodiment. In theembodiments so far, examples in which the internal parameters of theneural network can be adjusted have been described, but the embodimentsare not limited to these examples. For example, in the “learning mode”,the image processing system 100 generates neural networks according tothe amount of noise using training images classified according to theamount of noise. That is, as shown in FIG. 16 , in the “learning mode”,the image processing system 100 generates a plurality of neural networks60 f whose internal parameters have been adjusted according to theamount of noise σ_(m) (m=1 to M).

In the meantime, in the “image processing mode”, the image processingsystem 100 determines the amount of noise G_(m) of the input image basedon the SNR-related data acquired from the input image itself or from theadditional information of the input image by the SNR-related dataacquisition function. Further, the image processing system 100 selects,by the neural network selection function 500 and the selector 820, aneural network 60 f whose internal parameters have been adjusted byσ_(m) closest to the determined amount of noise σ. Using the selectedneural network 60 f, the image processing system 100 outputs an image inwhich noise is removed or reduced as compared with the input image.

FIG. 17 is a block diagram showing a configuration example of anembodiment in which the image processing system 100 is adapted to amedical information processing system. In the embodiments so far, theneural network has been described as an example of a machine learningmodel after training, but the embodiment is not limited to this. Themodel after training obtained by the machine learning is not limited toprocessing apparatuses in which one or more nodes imitating cells areconnected by one or more edges imitating nerves, but may be a functionhaving processing parameters. For example, a model after trainingobtained by the machine learning may be a regression model such as alinear regression model or a support vector machine (SVM) model, or atree model such as a decision tree model, a regression tree model, or arandom forest model. Incidentally, the model after training includes amodel updated by self-training.

In the embodiments so far, the term “input image” is used as a target tobe input to the neural network, but this “input image” includesprojection data acquired by the CT apparatus, or input data beforereconstruction such as k-space data obtained by the MRI apparatus. InFIG. 17 , a model after training 600 having processing parameters 600 a,600 b, and 600 c is illustrated. The input I/F 11 acquires input dataobtained by the medical image diagnostic apparatus. Further, the inputdata index acquisition function 700 acquires an index value related tonoise of input data or an index value indicating the property of theinput data. The processing parameter 600 b is a processing parametercorresponding to the acquired index value. The model after trainingprocessing function 710 controls the model after training 600 having theprocessing parameter 600 b corresponding to the index value in the inputdata so as to process the input data.

In the above-described embodiments, the processing circuitry or thecomputer constituting the image processing system 100 has been describedas being one. However, the number of processing circuitries or computersconstituting the image processing system 100 is not limited to one, andthe above-described processing may be performed by using pluralprocessing circuitries or plural computers.

According to at least one embodiment described above, even when theamount of noise superimposed on an image at the time of use is differentfrom the amount of noise of the training image in the image processingsystem that uses the neural network, high noise removal or high noisereduction performance can be secured.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel methods and systems describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the methods andsystems described herein may be made without departing from the spiritof the inventions. The accompanying claims and their equivalents areintended to cover such forms or modifications as would fall within thescope and spirit of the inventions.

What is claimed is: 1: A medical information processing systemcomprising: a memory configured to store a predetermined program; andprocessing circuitry configured, by executing the predetermined program,to (a) in a training phase, create a neural network by a machinelearning, wherein the neural network comprises an input layer, an outputlayer, and an intermediate layer provided between the input layer andthe output layer, wherein the intermediate layer includes at least onenode having at least one internal parameter, and the internal parameteris determined through the machine learning in the training phase; (b) inan interference phase conducted after the training phase, acquire inputdata obtained by imaging an object with a medical image diagnosticapparatus; input the input data into the neural network; determine amaximum pixel value in the input data, normalize the input data bydividing a predetermined value by the maximum pixel value to generate again value, set the gain value as a control signal, apply denoiseprocessing to the input data by adjusting the internal parameter of theneural network in accordance with the control signal. 2: The medicalinformation processing system according to claim 1, wherein the inputdata is an image that includes noise; and the processing circuitry isconfigured to perform the processing on the input data in such a mannerthat the noise is reduced or removed from the input data. 3: The medicalinformation processing system according to claim 1, wherein the controlsignal is generated from SNR-related data that are related to magnitudeof noise in the input data or signal-to-noise ratio of the input data.4: The medical information processing system according to claim 1,wherein the node has an activation function; and the internal parameteris a parameter of the activation function. 5: The medical informationprocessing system according to claim 4, wherein the activation functionis a function of removing such an input signal included in the inputdata that amplitude thereof is smaller than a threshold value; and theinternal parameter is the threshold value of the activation function. 6:The medical information processing system according to claim 4, whereinthe activation function is a function that outputs zero in a case ofreceiving an input value within a range including zero and outputs avalue proportional to an input value in a case of receiving an inputvalue outside the range. 7: The medical information processing systemaccording to claim 1, wherein the neural network includes a first neuralnetwork and a second neural network; and the processing circuitry isconfigured to adjust an internal parameter of the first neural networkbased on the data related to the input image, and does not adjust aninternal parameter of the second neural network. 8: The medicalinformation processing system according to claim 1, wherein the neuralnetwork comprises: a first neural network; a second neural network; adivider configured to divide the input data into a high-frequencycomponent image and a low-frequency component image, output thehigh-frequency component image to the first neural network, and outputthe low-frequency component image to the second neural network; and animage composer configured to compose a first output image outputted fromthe first neural network and a second output image outputted from thesecond neural network, and the processing circuitry is configured toadjust an internal parameter of the first neural network based on thedata related to the input data, and does not adjust an internalparameter of the second neural network. 9: The medical informationprocessing system according to claim 1, wherein, the processingcircuitry is configured to adjust the internal parameter based on animaging condition. 10: The medical information processing systemaccording to claim 1, wherein, the medical information processing systemis configured as at least one computer. 11: The medical informationprocessing system according to claim 1, wherein, the medical informationprocessing system is configured as at least one of an MRI apparatus, aCT apparatus, and an ultrasound diagnostic apparatus. 12: The medicalinformation processing system according to claim 1, wherein, theprocessing circuitry is configured to calculate the internal parameteraccording to a function having, as a variable, a value based on datarelated to the input data, each time the input data is input.